Medical image diagnostic apparatus, image processing apparatus, and registration method

ABSTRACT

A medical image diagnostic apparatus according to an embodiment includes processing circuitry configured to determine a plurality of small blocks for each of a plurality of pieces of medical image data, generate a plurality of superpixels corresponding to the plurality of small blocks, assign a label to at least one of the plurality of pieces of medical image data, and perform registration between the plurality of pieces of medical image data using the plurality of superpixels and the label.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromChinese Patent Application No. 201810670240.0, filed on Jun. 26, 2018;and Japanese Patent Application No. 2019-94382, filed on May 20, 2019,the entire contents of all of which are incorporated herein byreference.

FIELD

Embodiments described herein relate generally to a medical imagediagnostic apparatus, an image processing apparatus, and a registrationmethod.

BACKGROUND

For a single image obtained by scanning with a device such as a CT, anMR, or an ultrasonic scanner, in conventional medical image diagnosticapparatuses, there is a technique of obtaining an image of the entireblood vessel portion by manually labeling a tubular tissue such as ablood vessel.

However, for the same patient, when scanned with different devices, forexample, when both CT scan and ultrasound scan are performed, for thesame tissue, for example, the same blood vessel, sometimes there is theneed of comparing its features in images of different modalities (i.e.,in CT scan image and in ultrasonic scan image), so as to find a lesionor the like of the blood vessel. However, sometimes since there are manyblood vessels in the image, it is difficult for the user to find thecorresponding blood vessels in the above mentioned two scanned images,making it impossible to perform a rapid and accurate diagnosis.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the structure of a medical image diagnosticapparatus according to a first embodiment;

FIG. 2A is a schematic view of a superpixel segmentation performed on amedical image;

FIG. 2B is a schematic view of a superpixel segmentation performed on amedical image;

FIG. 3 is a structural diagram of a superpixel calculation unit;

FIG. 4A is a graph for illustrating registration processing betweenpieces of medical image data;

FIG. 4B is a graph for illustrating registration processing betweenpieces of medical image data;

FIG. 5 is a flowchart for illustrating registration processing betweenpieces of medical image data;

FIG. 6 is a flowchart for illustrating how to find a correspondingsuperpixel in a floating image;

FIG. 7A is a schematic diagram showing a state in which medical imagedata has been registered;

FIG. 7B is a schematic diagram showing a state in which medical imagedata has been registered;

FIG. 8 is a block diagram of the structure of a medical image diagnosticapparatus according to a second embodiment;

FIG. 9A is a diagram illustrative of the correction made when there isan error in the superpixel calculated in a floating image;

FIG. 9B is a diagram illustrative of the correction made when there isan error in the superpixel calculated in a floating image;

FIG. 10 is a flowchart of the registration processing of medical imagedata according to a second embodiment;

FIG. 11 is a block diagram of the structure of a medical imagediagnostic apparatus according to a third embodiment;

FIG. 12 is a flowchart of the registration processing of medical imagedata according to a third embodiment; and

FIG. 13 is a block diagram of the exemplary structure of a medical imagediagnostic system according to a fourth embodiment.

DETAILED DESCRIPTION

A medical image diagnostic apparatus comprises processing circuitry. Theprocessing circuitry is configured to determine a plurality of smallblocks for each of a plurality of pieces of medical image data, andgenerate a plurality of superpixels corresponding to the plurality ofsmall blocks; assign a label to at least one of the plurality of piecesof medical image data; and perform registration between the plurality ofpieces of medical image data, using the plurality of superpixels and thelabel.

In the following, embodiments of the medical image diagnostic apparatus,an image processing apparatus, and a registration method thereof will bedescribed with reference to the accompanying drawings.

In the following, a medical image diagnostic apparatus according to afirst embodiment will be described with reference to FIG. 1 to FIG. 7E.

FIG. 1 is a block diagram of the structure of a medical image diagnosticapparatus 100.

As shown in FIG. 1, the medical image diagnostic apparatus 100 includesa feature extraction unit 101, an image segmentation unit 102, alabeling unit 103, and a superpixel calculation unit 104. The imagesegmentation unit 102 is one example of a generating unit. The labelingunit 103 is one example of a labeling unit. The superpixel calculationunit 104 is one example of a registration unit.

The feature extraction unit 101, the image segmentation unit 102, thelabeling unit 103, and the superpixel calculation unit 104 may beimplemented by the processing circuitry. The processing circuitry isprocessor that implements the functions that correspond to therespective programs by reading the programs from memory and executingthem.

The feature extraction unit 101 extracts image features such asgrayscale value, gradient, shape, position, and Hessian matrix for amedical image selected by a user such as a doctor.

The image segmentation unit 102 is used for performing superpixelsegmentation on medical images selected by the user.

Superpixel segmentation is a process of subdividing an image intomultiple image sub-regions, i.e., superpixels, in order to locateobjects and boundaries, etc. in the image. A superpixel is a smallregion consists of a series of pixels whose positions are adjacent andwhose characteristics (such as color, brightness, and texture) aresimilar.

FIG. 2A and FIG. 2B are two images taken on the same patient, whereinthe left side image of FIG. 2A is an ultrasound image P1, and the leftside image of FIG. 2B is a CT image P2, in both images, the same vesselsare shown respectively, a blood vessel vs1 is shown in the ultrasonicimage P1, and a blood vessel vs2 which is the same as the blood vesselvs1 is shown in the CT image P2. A blood vessel is one example of atubular structure.

The graph on the right side of FIG. 2A is a superpixel segmentation mapS1 including the blood vessel vs1, which is an enlarged view of the boxpart in the image after a superpixel segmentation of the ultrasoundimage P1, and the graph on the right side of FIG. 2B is a superpixelsegmentation map S2 including the blood vessel vs2, which is an enlargedview of the box part in the image after a superpixel segmentation of theCT image P2. The superpixel segmentation map S1 and the superpixelsegmentation map S2 are each composed of a plurality of superpixels sp.

In FIG. 2A, only the Y-shaped blood vessel including the blood vesselvs1 is clearly shown, and in FIG. 2B, the Y-shaped blood vesselincluding the blood vessel vs2 is clearly shown, however, the figuresare chosen for the convenience of the description, and generally, aplurality of blood vessels will present in each image, in case the bloodvessel vs1 is located in the ultrasound image P1, it would be difficultto find the corresponding same blood vessel vs2 in the CT image P2 withnaked eye.

Further, in the present embodiment, the ultrasonic image P1 is used torepresent the reference image, and the CT image P2 is used to representthe floating image. By continuously labeling the superpixelsconstituting the blood vessel on the ultrasonic image P1 representativeof the reference image, it is possible to automatically display thesuperpixels of the corresponding vessel on the CT image P2representative of the floating image, thereby displaying thecorresponding blood vessel.

The labeling unit 103 receives a labeling applied by a user with afinger or an electronic pen or the like on a superpixel segmentation mapdisplayed on the touch panel, i.e., label assignment, and can highlightthe superpixels where the labeled point resides or the superpixels thelabeled line passes, that is, a plurality of superpixels correspondingto a plurality of small blocks determined within each medical image datacan be generated based on the assigned labels. The highlighting may bedistinguished from other superpixels by, for example, changing the colorof the superpixel or the like.

For the same component, such as the vessel in the reference image andthe floating image, the superpixel calculation unit 104 calculates thesuperpixels on the floating image that correspond to the superpixels onthe reference image in accordance with the plurality of superpixelshighlighted by continuous line labeling on the reference image and thestarting superpixel labeled on the floating image, and highlights it,thereby performing registration between the pieces of data of thereference image and the floating image.

FIG. 3 is a structure diagram showing the structure of the superpixelcalculation unit 104. The superpixel calculation unit 104 comprises acenter point calculation unit 1041, a matrix generation unit 1042, acoordinate conversion unit 1043, a selection unit 1044, a featureacquisition unit 1045, a similarity calculation unit 1046, a judgmentunit 1047, and a determination unit 1048.

The center point calculation unit 1041 can calculate the centroid of thesuperpixel as the center point of the superpixel for the superpixelhighlighted by the labeling unit 103.

FIG. 4A and FIG. 4B are diagrams for illustrating registrationprocessing between pieces of medical image data, in which FIG. 4Acorresponds to the superpixel segmentation map S1 in FIG. 2A, and FIG.4B corresponds to the superpixel segmentation map S2 in FIG. 2B. In FIG.4A and FIG. 4B, in order to make each label clearer, the grayscale inthe superpixel segmentation maps S1 and S2 are removed.

As shown in FIG. 4A and FIG. 4B, in the superpixel segmentation map S1,the center point calculation unit 1041 can calculate the center pointsof the superpixel sp1 and the superpixel sp2 obtained by labeling, andthe center points are represented by coordinates x1 and x2,respectively. In the superpixel segmentation map S2, the center pointcalculation unit 1041 can calculate the center point of the superpixelsp1′ obtained by labeling, and the center point is represented by thecoordinate y1.

The matrix generation unit 1042 generates a transformation matrix Tbetween the blood vessel vs1 of the superpixel segmentation map S1 andthe blood vessel vs2 of the superpixel segmentation map S2 in accordancewith the coordinates of the center points of the plurality ofsuperpixels (at least two superpixels) of the blood vessel vs1 of thesuperpixel segmentation map S1 and the coordinates of the center pointsof the superpixels of the blood vessel vs2 of the segmentation map S2corresponding to the plurality of superpixels of blood vessel vs1. Thetransformation matrix T is, for example, an affine transformationmatrix, which can be calculated, for example, by a least squares method.

The coordinate conversion unit 1043 multiplies the coordinate x2 withthe transformation matrix T to obtain the predicted coordinate C (x2) ofthe superpixel sp2′ of the superpixel segmentation map 52.

The selection unit 1044 selects superpixels located completely or partlywithin the range of a region with a radius r centered on the coordinateC (x2) respectively, as shown in FIG. 4A and FIG. 4B, the superpixelsspx′, spy′, spz′, sp2′ are selected, respectively. Of course, the scaleof the radius r can be set as needed.

The feature acquisition unit 1045 acquires the respective features (suchas grayscale value, gradient, shape, location, and Hessian matrix, etc.)of the features extracted from the feature extraction unit 101 for eachof the superpixels spx′, spy′, spz′, sp2′.

For example, 1-d features of the superpixels are obtained, whenobtaining the features of superpixel spx′, the features are representedby a set a_(j)=(a_(j1), a_(j2), . . . , a_(jd)). Meanwhile, for thesuperpixel sp2 in the superpixel segmentation map S1, the featureacquisition unit 1045 obtains its feature a₂=(a₂₁, a₂₂, . . . , a_(2d)).

The similarity calculation unit 1046 calculates the similarity of thefeatures between the superpixel sp2 in the superpixel segmentation mapS1 and the superpixel spx′ in the superpixel segmentation map S2,specifically, puts the features a_(j) and a₂ into one set, i.e., forminga set ak=(a₂, a_(j))=(a₂₁, a₂₂, . . . , a_(2d), a_(j1), a_(j2), . . . ,a_(jd)), and compares each feature, to calculate the similarity s_(2j)of the overall feature a_(j) and a₂.

The judgment unit 1047 judges whether or not all superpixels within therange of the radius r centered on the coordinate C (x2) in thesuperpixel segmentation map S2 of the CT image P2 are selected.

By comparing the feature of each of the superpixels spx′, spy′, spz′,and sp2′ with the feature of the superpixel sp2, the determination unit1048 determines the superpixel having the closest similarity (thehighest similarity) as the superpixel in the CT image P2 thatcorresponds to the superpixel sp2.

In the following, the registration processing of the medical image datawill be described based on FIG. 2A and FIG. 2B, FIG. 4A and FIG. 4B, andFIG. 5, wherein FIG. 5 is a flowchart for illustrating the registrationprocessing of the medical image data.

First, the user finds two images taken on the same patient, i.e., anultrasound image P1 (reference image) and a CT image P2 (floatingimage). And in the two images, the user judges that the same bloodvessel (i.e., the blood vessel vs1 in the ultrasound image P1 and theblood vessel vs2 in the CT image P2) should exist by a comparison ofimage slice position. That is, the user obtains the reference image andthe floating image in which the same blood vessel exists (step S11).

After that, the feature extraction unit 101 extracts image features suchas grayscale value, gradient, shape, position, and Hessian matrix forthe ultrasound image P1 and the CT image P2 (step S12).

After that, as shown in the superpixel segmentation map S1 and thesuperpixel segmentation map S2 in the right portion of FIG. 2A and FIG.2B, the image segmentation unit 102 performs superpixel segmentation onthe ultrasound image P1 and the CT image P2 selected by the user (stepS13).

After that, as shown in FIG. 4B, in the superpixel segmentation map S2,that is, in the CT image P2, the user performs manual labeling bytouching the position of the superpixel sp1′ with a finger or anelectronic pen or the like, and the labeling unit 103 receives theuser's labeling, i.e., label assignment, thereby highlighting thesuperpixel sp1′ where the labeled point resides, that is, generating thesuperpixel sp1′ (step S14).

After that, as shown in FIG. 4A, in the superpixel segmentation map S1,that is, in the ultrasound image P1, using a finger or an electronic penor the like, along the extending direction of the blood vessel vs1,taking the position of the superpixel sp1 that corresponds to thesuperpixel sp1′ as the starting point, the user draws a linecontinuously, thereby manually labeling the blood vessel vs1. Thelabeling unit 103 receives the user's labeling, that is, labelassignment, thereby highlighting the superpixels sp1, sp2, etc. wherethe labeled line is located, and the blood vessel vs1 is graduallydisplayed (step S15).

After that, in the superpixel segmentation map S2 (i.e., the CT imageP2), the superpixel calculating unit 104 calculates superpixelscorresponding to the plurality of superpixels of the ultrasound image P1in accordance with the plurality of superpixels highlighted by labelingby the continuous drawing a line on the superpixel segmentation map S1(i.e., the ultrasonic image P1) and the starting pixel labeled on thesuperpixel segmentation map S2 (i.e., the CT image P2), andautomatically labeling is performed by highlighting (step S16).

In the following, a method of calculating corresponding superpixels inthe CT image P2 will be described based on FIG. 4A, FIG. 4B, and FIG. 6,wherein FIG. 6 is a flowchart for illustrating how to find acorresponding superpixel in a floating image.

First, as shown in FIG. 4A, for the superpixel segmentation map S1, thecenter point calculation unit 1041 calculates the superpixel sp2 labeledby the continuous drawing the line and the coordinates the center pointsx1 and x2 of the superpixel sp1. For the superpixel segmentation map S2,the center point calculation unit 1041 calculates the center pointcoordinate y1 of the superpixel sp1′ obtained by the labeling. And thematrix generation unit 1042 generates transformation matrix T betweenthe blood vessel vs1 of the superpixel segmentation map S1 and the bloodvessels vs2 of the superpixel segmentation map S2 in accordance with thecoordinates of the center points x1, x2, and y1 (step S161).

After that, as shown in FIG. 4B, in the superpixel segmentation map S2(i.e., the CT image P2), the coordinate conversion unit 1043 multipliesthe coordinate x2 with the transformation matrix T to obtain thepredicted coordinate C(x2) of the superpixel sp2′ of the superpixelsegmentation map S2 (step S162).

After that, as shown in FIG. 4A, in the superpixel segmentation map S1(i.e., the ultrasound image P1), the feature acquisition unit 1045acquires the features of the later labeled (i.e., the newest labeled)superpixel sp2 in the line labeling, a₂=(a₂₁, a₂₂, . . . , a_(2d)) (stepS163).

After that, as shown in FIG. 4B, in the superpixel segmentation map S2(i.e., the CT image P2), within the range of the radius r centered onthe coordinate C (x2), the selection unit 1044 selects the superpixelsthat are partly within the range. For the superpixel spx′, the featureacquisition unit 1045 obtains the feature a_(j)=(a_(j1), a_(j2), . . . ,a_(jd)) from the feature extracted by the feature extraction unit 101(step S164).

After that, the similarity calculation unit 1046 puts the features a_(j)and a₂ into one set, that is, forming the set ak=(a₂, a_(j))=(a₂₁, a₂₂,. . . , a_(2d), a_(j1), a_(j2), . . . , a_(jd)), and compares eachfeature to calculate the similarity s_(2j) of the whole a₂ and a_(j),thereby calculating a similarity between the features of the superpixelsp2 in the superpixel segmentation map S1 and the superpixel spx′ in thesuperpixel segmentation map S2 (step S165).

After that, the judgment unit 1047 judges whether all superpixels withinthe range of the radius r centered on the coordinate C (x2) are selected(step S166). In case it is judged that not all the superpixels areselected (NO in step S166), the method returns to step S164, and asshown in FIG. 4B, in the superpixel segmentation map S2, superpixelspy′, spz′ or sp2′ are selected and their features are obtained, andthen proceeding to step S165. In case it is determined superpixels spx′,spy′, spz′, sp2′ are all selected (YES in step S166), by comparing thesimilarity of the features of each of the superpixels spx′, spy′, spz′,sp2′ with the features of the superpixel sp2, the determination unit1048 determines the superpixel having closest similarity as thecorresponding superpixel in the CT image P2 (step S167). Here, since thegrayscale of superpixel sp2′ is most similar to that of superpixel sp2,both are blood vessels, and the shapes are also similar and so on,therefore, in the CT image P2, the superpixel sp2′ is determined as thesuperpixel that corresponds to the superpixel sp2 in the ultrasonicimage P1.

FIG. 7A and FIG. 7B are diagrams showing a state in which medical imagedata has been registered. In FIG. 7B, in the superpixel segmentation mapS2 in the CT image P2, in case a certain point of the region where thesuperpixel sp1′ (starting superpixel) resides is labeled, in FIG. 7A, inthe superpixel segmentation map S1 in the ultrasound image P1, when aline is continuously drawn from a certain point of the region where thesuperpixel sp1 (starting superpixel) resides to a certain superpixel inthe region where the superpixel sp6 resides along the direction of theblood vessel vs1, as is shown in FIG. 7B, it can be automaticallylabeled to the superpixel sp6′ in the superpixel segmentation map S2 inthe CT image P2, thereby automatically finding the blood vessel vs2corresponding to blood vessel vs1. Therefore, the user can easily findthe same blood vessel in the two scanned images of the ultrasonic imageP1 and the CT image P2, thereby enabling rapid and accurate diagnosis.

Further, in the superpixel calculation unit 104, the feature acquisitionunit 1045 and the judgment unit 1047 may not be provided, and when thefeature acquisition unit 1045 is not provided, the feature extractionunit 101 provides the feature of the superpixel, and when the judgmentunit 1047 is not provided, Step S166 in FIG. 6 is omitted, and in thiscase, in step S164, as shown in FIG. 4B, in the superpixel segmentationmap S2 (i.e., the CT image P2), within the range of a radius r centeredon the coordinate C (x2), the selection unit 1044 selects all thesuperpixels spx′, spy′, spz′, and sp2′ that are partly within the range.And the features of each superpixel are provided by the featureextraction unit 101. Thereafter, in step S165, the similaritycalculation unit 1046 calculates the similarity between the features ofthe superpixel sp2 in the superpixel segmentation map S1 and thesuperpixels spx′, spy′, spz′ and sp2′ in the superpixel segmentation mapS2. After that, at step S167, the determination unit 1048 determines thesuperpixel in superpixels spx′, spy′, spz′, sp2′ with features mostsimilar to the superpixel sp2 as the corresponding superpixel in CTimage P2.

In addition, as described above, the method of calculating thecorresponding superpixel is illustrated by taking FIG. 4A and FIG. 4B asan example. In FIG. 7A and FIG. 7B, in calculating the superpixel sp6′,the transformation matrix T between the blood vessel vs1 and the bloodvessel vs2 is calculated in accordance with the coordinates x1, x2, x3,x4, x5 of the center point of respective superpixels of the blood vesselvs1 in the superpixel segmentation S1 and the coordinates y1, y2, y3,y4, y5 of the center point of the superpixel of the blood vessel vs2 inthe superpixel segmentation map S2 corresponding to the coordinates ofthe center points of the above-described respective superpixels. And thepredicted coordinate of the center point of the superpixel sp6′ iscalculated by multiplying the center point coordinate x6 of thesuperpixel sp6 with the transformation matrix T, and the calculationmethod of the superpixel sp6′ thereafter resembles the steps S163-stepS167 in FIG. 6.

In the following, a medical image diagnostic apparatus 200 according toa second embodiment will be described with reference to FIG. 8 to FIG.10.

Moreover, only the distinguishing parts that differ from the firstembodiment will be described, and parts that are the same as those inthe first embodiment are given corresponding reference numbers, to avoidrepeated description or simplify the description.

FIG. 8 is a structure block diagram of the medical image diagnosticapparatus 200, FIG. 9A and FIG. 9B are illustrative diagrams ofcorrecting a superpixel error calculated in the superpixel segmentationmap S2 of the CT image, and FIG. 10 is a flowchart of the registrationprocessing of the medical image data.

As shown in FIG. 8, the medical image diagnostic apparatus 200 includesa feature extraction unit 201, an image segmentation unit 202, alabeling unit 203, a superpixel calculation unit 204 and a correctioninput unit 205.

Wherein the functions of the feature extraction unit 201, the imagesegmentation unit 202, the labeling unit 203, and the superpixelcalculation unit 204 are the same as the feature extraction unit 101,the image segmentation unit 102, the labeling unit 103, and thesuperpixel calculation unit 104 in the first embodiment, and therefore,a simplified description is made.

The feature extraction unit 201 extracts image features such as graysale value, gradient, shape, position, and Hessian matrix for a medicalimage selected by a user such as a doctor.

The image segmentation unit 202 is used for performing superpixelsegmentation on medical images selected by the user.

The labeling unit 203 receives a labeling applied by a user with afinger or an electronic pen or the like on a superpixel segmentation mapdisplayed on the touch panel, i.e., label assignment, and can highlightthe superpixel where the labeled point resides or the superpixel wherethe labeled line passes.

In accordance with the plurality of superpixels highlighted by labelingon the reference image by continuous drawing lines, the superpixelcalculation unit 204 calculates the corresponding superpixels on thefloating image, and highlights them.

In response to the calculation result (registration result) of thesuperpixel calculation unit 204, the user (operator) judges whether thecalculation result is appropriate, and if it is determined to beinappropriate, the correction input unit 205 receives the correctionindication made by the user via label assignment.

In the following, the functions of the correction input unit 205 will bedescribed based on FIG. 9A and FIG. 9B.

In FIG. 9B, in the superpixel segmentation map S2 in the CT image, incase a certain point of the region where the superpixel sp1′ (startingsuperpixel) resides is labeled, the superpixel sp1′ is generated by thelabeling unit 203. And in FIG. 9A, in the superpixel segmentation map S1in the ultrasound image, when continuously drawing a line from a certainpoint of the region where the superpixel sp1 (starting superpixel)resides, along the direction of the blood vessel vs1, to the certainpoint of the region where the superpixel sp4 resides via the superpixelssp2, sp3, for the superpixels sp2, sp3, the superpixel calculation unit204 calculates in the blood vessel vs2 of the superpixel segmentationmap S2 its corresponding superpixel as sp2′ and sp3′, and for thesuperpixel sp4, the superpixel calculation unit 204 calculates itscorresponding superpixel as sp0′. And at this time instant, the userrealizes that the calculation result is inappropriate (i.e., erroneous),he touches the display screen with a finger or an electronic pen or thelike to drag the superpixel sp0′ to the correct superpixel sp4′, or heclicks on the superpixel sp0′ first and then clicks on the superpixelsp4′ or in some other way to assign a label for correction indication,the correction input unit 205 receives the correction indication fromthe user.

In the following, the registration processing of medical image data willbe illustrated based on FIG. 9A, FIG. 9B, and FIG. 10.

Wherein steps S21 to S26 are the same as steps S11 to S16 of the firstembodiment, and thus repeated descriptions will be omitted.

In step S27, if the user judges that the calculation result of thesuperpixel calculation unit 204 is correct (YES in step S27), the methodreturns to step S25, the action of continuously manually labeling theblood vessel in the ultrasound image (i.e., the reference image)continues; and if the user judges that the calculation result of thesuperpixel calculation unit 204 is inappropriate, i.e., if it iserroneous (NO in step S27), the user assigns a label to make thecorrection indication, and the correction input unit 205 receives thecorrection indication from the user, and based on the label after thecorrection indication, the labeling unit 203 generate the superpixel insuch a way that it highlights the superpixel in which the label resides.And as is shown in FIG. 9B, the erroneous superpixel sp0′ is correctedto the correct superpixel sp4′, that is, the corresponding correctsuperpixel is manually selected (step S28). And after that, the methodreturns to step S25 to continue the action of continuously manuallylabeling the blood vessel in the ultrasonic image, i.e., the referenceimage.

Therefore, in case the calculation result of the superpixel calculationunit 204 is erroneous, the erroneous superpixel can be manuallycorrected to the correct superpixel, so that the user can easily find inthe CT image (i.e., the floating image) the blood vessel vs2 that is thesame as the blood vessel vs1 in the ultrasound image (i.e., thereference image), thereby enabling rapid and accurate diagnosis.

In the following, a medical image diagnostic apparatus according to athird embodiment will be described with reference to FIG. 11 to FIG. 12.

Moreover, only the distinguishing parts that differ from the secondembodiment will be described, and parts that are the same as those inthe second embodiment are given corresponding reference numbers, to omitrepeated descriptions.

FIG. 11 is a structure block diagram of a medical image diagnosticapparatus 300 of the third embodiment. FIG. 12 is a flowchart of theregistration processing of medical image data.

The medical image diagnostic apparatus 300 includes a feature extractionunit 301, an image segmentation unit 302, a labeling unit 303, asuperpixel calculation unit 304, a correction input unit 305 and atraining unit 306.

Wherein the functions of the feature extraction unit 301, the imagesegmentation unit 302, the labeling unit 303, the superpixel calculationunit 304 and the correction input unit 305 are the same as the featureextraction unit 201, the image segmentation unit 202, the labeling unit203, the superpixel calculation unit 204 and the correction input unit205 in the second embodiment, and therefore, a simplified description ismade.

The training unit 306 trains the superpixel calculation unit 304 in thefollowing way: storing the correction indication received by thecorrection input unit 305 and the calculation result of the superpixelcalculated by the superpixel calculation unit 304, assigning highweights to the correct calculation result of the superpixel, andassigning low weights to the erroneous calculations indicated to becorrected. As the training process is iterated, thereafter, theprobability that the correct superpixels are selected gets higher andhigher.

In the following, the registration processing of medical image data willbe illustrated based on FIG. 12.

Wherein steps S31 to S35, step S37, and step S38 are the same as stepsS21 to S25, step S27, and step S28 of the second embodiment, andtherefore repeated descriptions will be omitted.

In step S39, the training unit 306 trains the superpixel calculationunit 304 in accordance with the correction indication received by thecorrection input unit 305 and the calculation result of the correctsuperpixel calculated by the superpixel calculation unit 304, and thetraining result is reflected in the later generation of the superpixels,in order to generate the correct superpixels in the subsequentgeneration of the superpixels.

In step S36, the superpixel calculation unit 304 refers to the previoustraining result obtained by the training made to the superpixelcalculation unit 304 by the training unit 306, and calculates in the CTimage (i.e., the floating image) the superpixel corresponding to thesuperpixel in the ultrasound image (i.e., the reference image), andautomatically labels the superpixel.

In the above embodiment, by training the superpixel calculation unit 304by the training unit 306, the calculation result of the superpixel ofthe superpixel calculation unit 304 can be made more accurate, so thatthe user can perform a more rapid and accurate diagnosis.

Moreover, in the above-described embodiment, it can also be constructedas, the superpixel calculation unit 304 is provided with a learning unitthat learns the correction indication received by the correction inputunit 305 and reflects the learning result in the later generation of thesuperpixel by the superpixel calculation unit 304 of the medical imagediagnostic apparatus 300.

In the above embodiment, an ultrasound image and a CT image are taken asan example to perform registration between their pieces of data.However, images obtained by other imaging modalities are possible.Further, for convenience of explanation, nominations such as referenceimage and floating image are used, but other nominations can be used aslong as they are multiple images taken on the same patient.

The first to the third embodiments are explained above, but, in additionto those embodiments described above, implementations in variousdifferent forms are still possible.

For example, explained in the above embodiments is the registrationbetween two images, but the configuration is also applicable toregistration between three or more images. In the following, an exampleof registration between an ultrasound image, a CT image, and an MR imagewill be explained.

For example, first, the feature extraction unit 101, the featureextraction unit 201, or the feature extraction unit 301 (hereinafter,simply referred to as a feature extraction unit) extracts image featuresfrom each of the ultrasound image, the CT image, and the MR image. Theimage segmentation unit 102, the image segmentation unit 202, or theimage segmentation unit 302 (hereinafter, simply referred to as an imagesegmentation unit) then determines a plurality of small blocks for eachof the ultrasound image, the CT image, and the MR image based on theextracted image features, and generates a plurality of superpixelscorresponding to the determined plurality of small blocks.

The labeling unit 103, the labeling unit 203, or the labeling unit 303(hereinafter, simply referred to as a labeling unit) then assigns alabel to the medical image data of at least one of the ultrasound image,the CT image, and the MR image. For example, the labeling unit assigns alabel to the ultrasound image by receiving an input operation from anoperator.

The superpixel calculation unit 104, the superpixel calculation unit204, or the superpixel calculation unit 304 (hereinafter, simplyreferred to as a superpixel calculation unit) then performs registrationbetween the ultrasound image, the CT image, and the MR image, based onthe plurality of superpixels generated by the image segmentation unit,and on the label assigned in the ultrasound image. For example, first,the superpixel calculation unit determines a superpixel corresponding tothe superpixel to which the label is assigned in the ultrasound image,in the CT image. The superpixel calculation unit then performsregistration between the ultrasound image and the CT image, based on thesuperpixel to which the label is assigned in the ultrasound image, andon the superpixel determined in the CT image. The superpixel calculationunit also determines a superpixel corresponding to the superpixel towhich the label is assigned in the ultrasound image, in the MR image.The superpixel calculation unit then performs registration between theultrasound image and the MR image, based on the superpixel to which thelabel is assigned in the ultrasound image, and on the superpixeldetermined in the MR image. By performing the registration between theultrasound image and the CT image and the registration between theultrasound image and the MR image, registration between the CT image andthe MR image is also achieved.

Further, explained in in the above embodiment is an example in which thelabeling unit assigns a label to a tubular structure of a subjectincluded in at least one of the pieces of medical image data byreceiving an input operation from an operator. However, the embodimentis not limited thereto, and the labeling unit may also be configured toperform the label assignment automatically.

For example, in the registration between the ultrasound image and the CTimage, the labeling unit assigns a label to the ultrasound image. Here,the labeling unit may assign the label by receiving an input operationfrom the operator, or automatically. To explain one example, thelabeling unit extracts an end of the blood vessel represented in theultrasound image by running a pattern matching on the ultrasound image,and assigns a label to the superpixel positioned at the extracted end.The superpixel calculation unit then performs registration between theultrasound image and the CT image based on the label automaticallyassigned to the ultrasound image.

Further, explained in the above embodiments is an example in which alabel is assigned to a superpixel after the superpixels are generated.However, the order of the superpixel generation and the label assignmentmay be any order. For example, the superpixel generation and the labelassignment may be performed in parallel. It is also possible to generatesuperpixels after the label is assigned to the medical image data.

Further, when the label assignment is to be performed first, thesuperpixel generation may be performed based on the assigned label. Forexample, in the registration between the ultrasound image and the CTimage, first, the labeling unit assigns a label to the ultrasound imageby receiving an input operation from the operator, or automatically. Inother words, the labeling unit assigns a label to the ultrasound imageprior to the superpixel segmentation. For example, the labeling unitassigns a line drawn on the ultrasound image as a label.

The image segmentation unit then performs a superpixel segmentation onthe ultrasound image based on the assigned label. For example, first,the image segmentation unit generates a small region having its centroidon the line drawn in the ultrasound image, and consisting of a group ofadjacent pixels having similar image features, as a superpixel. Theimage segmentation unit then segments the entire ultrasound image intosuperpixels, by sequentially generating a small region adjacent to thegenerated superpixel, and consisting of a group of adjacent pixelshaving similar image features, as a superpixel. The image segmentationunit also determines a plurality of small blocks in the CT image basedon the image features, and generates a plurality of superpixelscorresponding to the determined plurality of small blocks. Thesuperpixel calculation unit then performs registration between theultrasound image and the CT image, based on the plurality of superpixelsgenerated for the ultrasound image and the CT image, and on the labelassigned to the ultrasound image.

Further, explained in the embodiments described above is an example inwhich the registration is performed by a medical image diagnosticapparatus, such as the medical image diagnostic apparatus 100, themedical image diagnostic apparatus 200, and the medical image diagnosticapparatus 300. However, the embodiment is not limited thereto.

For example, the medical image diagnostic apparatus 100 may be includedin a medical image diagnostic system 1 shown in FIG. 13, and an imageprocessing apparatus 400 connected to the medical image diagnosticapparatus 100 via a network NW may perform the registration. FIG. 13 isa block diagram of an exemplary structure of a medical image diagnosticsystem 1 according to a fourth embodiment. The image processingapparatus 400 is implemented as a computer device such as a workstation,for example.

FIG. 13 shows an example in which the image processing apparatus 400 isconnected to the medical image diagnostic apparatus 100, but the imageprocessing apparatus 400 may be connected to the medical imagediagnostic apparatus 200 or to the medical image diagnostic apparatus300, instead of the medical image diagnostic apparatus 100. Further,although one medical image diagnostic apparatus is shown in FIG. 13, itis also possible for the medical image diagnostic system 1 to include aplurality of medical image diagnostic apparatuses.

For example, the image processing apparatus 400 has functionscorresponding to those of the feature extraction unit, the imagesegmentation unit, the labeling unit, and the superpixel calculationunit described above, and performs registration between a plurality ofpieces of medical image data. To explain one example, the imageprocessing apparatus 400 obtains a CT image from a medical imagediagnostic apparatus that is a CT scanner, and obtains an ultrasoundimage from a medical image diagnostic apparatus that is an ultrasonicscanner, over the network NW.

Alternatively, the image processing apparatus 400 may acquire themedical image data from an image storage apparatus 500 over the networkNW. For example, the image storage apparatus 500 stores therein a CTimage acquired from a medical image diagnostic apparatus that is a CTscanner, or stores the CT image in a memory provided external to theapparatus. The image storage apparatus 500 also stores therein anultrasound image acquired from a medical image diagnostic apparatus thatis an ultrasonic scanner, or stores the ultrasound image in a memoryprovided external to the apparatus. The image processing apparatus 400then obtains the CT image and the ultrasound image from the imagestorage apparatus 500 over the network NW. The image storage apparatus500 is implemented as a computer device such as a server device, forexample.

The image processing apparatus 400 then determines a plurality of smallblocks for each of the ultrasound image and the CT image, and generatesa plurality of superpixels corresponding to the determined plurality ofsmall blocks. The image processing apparatus 400 also assigns a label toat least one of the ultrasound image and the CT image. The imageprocessing apparatus 400 then performs registration between theultrasound image and the CT image based on the generated plurality ofsuperpixels, and on the label assigned to the at least one of theultrasound image and the CT image.

Further, explained in the above embodiments is an example in which theregistration is performed by finding a tubular structure such as a bloodvessel in a plurality of pieces of medical image data, but the region tobe used in the registration is not limited to a tubular structure. Inother words, the above embodiments may be applied in the same mannereven when the registration is performed by finding another region such abone or a soft tissue of the subject.

Further, the registration method explained in the first to the fourthembodiment may be implemented by causing a computer, such as a personalcomputer or a work station, to execute a processing program prepared inadvance. This processing program may be provided over a network such asthe Internet. Further, this processing program may be recorded in acomputer-readable non-transitory recording medium such as a hard disk, aflexible disk (FD), a compact disc read-only memory (CD-ROM), amagneto-optical (MO) disc, or a digital versatile disc (DVD), andexecuted by causing a computer to read the processing program from therecording medium.

According to at least one of the embodiments explained above, it ispossible to find the same corresponding structure easily in a pluralityof images taken of the same patient by a plurality of imagingapproaches, and to achieve registration between images of differentmodalities, thereby enabling rapid and accurate diagnosis.

As described above, the embodiments have been described by way ofexample only, and are not intended to limit the scope of the invention.The present invention can be implemented in various other forms, andvarious omissions, substitutions and changes can be made withoutdeparting from the spirit of the invention. These embodiments and theirmodifications are included in the scope and spirit of the invention, andare included in the invention recited in the technical solution and thescope of its equivalents.

What is claimed is:
 1. A medical image diagnostic apparatus comprisingprocessing circuitry configured to: determine a plurality of smallblocks for each of a plurality of pieces of medical image data, andgenerate a plurality of superpixels corresponding to the plurality ofsmall blocks; assign a label to at least one of the plurality of piecesof medical image data; and perform registration between the plurality ofpieces of medical image data, using the plurality of superpixels and thelabel.
 2. The medical image diagnostic apparatus according to claim 1,wherein the processing circuitry is configured to assign the label to atleast one of the superpixels in at least one of the plurality of piecesof medical image data.
 3. The medical image diagnostic apparatusaccording to claim 1, wherein the processing circuitry is configured todetermine a plurality of small blocks based on the label, for the pieceof medical image data to which the label is assigned, among theplurality of pieces of medical image data, and to generate a pluralityof superpixels corresponding to the plurality of small blocks.
 4. Themedical image diagnostic apparatus according to claim 1, wherein theprocessing circuitry is configured to assign the label to at least oneof the plurality of pieces of medical image data by receiving an inputoperation from an operator.
 5. The medical image diagnostic apparatusaccording to claim 1, wherein the plurality of pieces of medical imagedata include first image data and second image data, and the processingcircuitry is configured to: generate a plurality of superpixels for eachof the first image data and the second image data; assign the label atleast to the first image data; determine a second superpixel in thesecond image data, the second superpixel being a superpixelcorresponding to a first superpixel to which the label is assigned inthe first image data; and perform registration between the first imagedata and the second image data based on the first superpixel and thesecond superpixel.
 6. The medical image diagnostic apparatus accordingto claim 1, wherein the processing circuitry is also configured to:receive a correction indication issued by an operator via assignment ofthe label, in response to a result of the registration; and regenerate asuperpixel based on the label assigned as the correction indication. 7.The medical image diagnostic apparatus according to claim 6, wherein theprocessing circuitry is configured to learn the received correctionindication, and to reflect result of the learning to a subsequentgeneration of the superpixels.
 8. The medical image diagnostic apparatusaccording to claim 1, wherein the processing circuitry is configured toassign the label to a region of a subject included in at least one ofthe plurality of pieces of medical image data.
 9. The medical imagediagnostic apparatus according to claim 8, wherein the region is atubular structure.
 10. The medical image diagnostic apparatus accordingto claim 9, wherein the tubular structure is a blood vessel.
 11. Themedical image diagnostic apparatus according to claim 8, wherein theprocessing circuitry is configured to: calculate a centroid of asuperpixel as a center point of the superpixel; generate atransformation matrix between a first region in first image data and asecond region in second image data, in accordance with coordinates ofcenter points of at least two superpixels of the first region, and withcoordinates of center points of corresponding superpixels of the secondregion, the corresponding superpixels being superpixels corresponding tothe at least two superpixels; generate predicted coordinates of a centerpoint of a second superpixel of the second region, the second superpixelbeing a superpixel corresponding a first superpixel of the first region,in accordance with coordinates of a center point of the first superpixeland the transformation matrix; select a plurality of superpixels withina range of a predetermined radius centered on the predicted coordinates;calculate a similarity between a feature of the first superpixel of thefirst region and a feature of each of the plurality of superpixelsselected within the range of the predetermined radius centered on thepredicted coordinates; and determine a superpixel exhibiting a mostsimilar feature in the second region, as the second superpixel, amongthe plurality of superpixels selected within the range of thepredetermined radius centered on the predicted coordinates.
 12. An imageprocessing apparatus comprising a processing circuitry configured to:determine a plurality of small blocks for each of a plurality of piecesof medical image data, and generate a plurality of superpixelscorresponding to the plurality of small blocks; assign a label to atleast one of the plurality of pieces of medical image data; and performregistration between the plurality of pieces of medical image data,using the plurality of superpixels and the label.
 13. A registrationmethod comprising: determining a plurality of small blocks for each of aplurality of pieces of medical image data, and generating a plurality ofsuperpixels corresponding to the plurality of small blocks; assigning alabel to at least one of the plurality of pieces of medical image data;and performing registration between the plurality of pieces of medicalimage data, using the plurality of superpixels and the label.